Advanced analytic methods and systems utilizing trust-weighted machine learning models

ABSTRACT

Systems and methods of the present disclosure include determining a performance status of a selected component in an aircraft. An ensemble of related machine learning models is applied to feature data extracted from flight data of the aircraft. Each model produces a positive score and a complementary negative score related to performance of the selected component. The positive scores are weighted based on the false positive rates of the models and the negative scores are weighted based on the false negative rates of the models. The weighted positive scores are combined, e.g., by averaging, and the weighted negative scores are combined, e.g., by averaging. The performance status of the selected component is determined as one of a positive category, a negative category, or an unclassified category based on the values of the combined weighted positive scores and the combined weighted negative scores.

FIELD

The present disclosure relates to advanced analytic methods and systemsutilizing trust-weighted machine learning models.

BACKGROUND

Aircraft and other complex apparatuses include a myriad ofinteroperating components. Many subsystems of components are designedfor maintenance and/or repair. When such a subsystem or componentperforms unexpectedly (e.g., it becomes non-responsive or functions withdegraded performance), the operation of the aircraft may be impacted andthe aircraft may be subject to unscheduled maintenance and down time. Asan example, servomechanisms operating flight control surfaces ofaircraft may become non-operational and may consequently result instrain on the flight control system, fuel waste, aircraft down time,excess troubleshooting, strain on the replacement part supply, and/orpotential stress on other aircraft components and systems. Theconsequences of an unexpected need to repair or an unexpected repair ofthe non-performing component may be much greater than the cost, in timeand resources, to repair or replace the non-performing component.

For many components, operators currently have no insight into the healthof the components. Moreover, subsystems and components may behaveerratically and unexpectedly well before complete non-performance. Thebehavior of components that may lead to non-performance may manifest asmerely non-specific system degradation and related effects.

Aircraft may be available for maintenance and diagnosis only betweenflights. If a component performs unexpectedly during a flight, thenon-performing component may require an abrupt end to the flight,necessitating a return trip for repair or at least temporary abandonmentof the aircraft.

If a component does not pass pre-flight tests, the component may need tobe repaired or replaced before the next flight. After a component hasperformed unexpectedly during a flight and resulted in noticeable systemperformance degradation, maintenance staff may troubleshoot the aircraftfunction and eventually identify the non-performing part as acontributor to the observed system performance degradation. Theunscheduled down time and maintenance to identify the issue and torepair or replace the non-performing component may lead to resourceconflicts with the aircraft. The aircraft typically is unavailable foruse during troubleshooting and repair. Additionally, the unscheduleddown time and maintenance may lead to strains on the scheduling ofmaintenance personnel due to excess time troubleshooting and identifyingthe issue, and may lead to strains on the supply chain for replacementparts because the need for parts may not be predictable. This reactiveresponse to non-performing components may be inefficient when comparedto scheduled maintenance or a proactive response to impending componentnon-performance.

Further, aircraft may include components designed to last the servicelife of the aircraft and not specifically designed for maintenanceand/or repair. For example, airframe components generally are expectedto function for the service life of the aircraft. However, somecomponents may not react to the stresses of use and the environment asexpected and some aircraft may be used beyond the originally designedservice life. In such cases, repair or replacement of structuralcomponents not originally designed to be repaired or replaced may causesignificant downtime for individual aircraft while the affectedstructural components are repaired or reproduced for replacement.

For example, the F/A-18 Hornet model of aircraft was first placed intooperational service in 1983. Now more than 30 years later, the majorityof F/A-18 Hornet aircraft in service are operated at or beyond theiroriginally designed service life (6,000-8,000 hours). Continuedoperation relies on a structured approach to inspection, maintenance,and repair that includes airframe repair and replacement. Airframeinspection, repair, and replacement are performed during cycles of heavymaintenance. During heavy maintenance, the airframe and other structuralcomponents are inspected for mechanical wear, heat damage, corrosion,and other signs of component fatigue. Though heavy maintenance commonlyresults in repair or replacement of some structural components,predicting which components will need repair or replacement in aparticular aircraft is very difficult with current technology. Hence,maintaining the F/A-18 Hornet fleet in serviceable condition leads tonew and variable demand for a large number of airframe and otherstructural components that were not originally designed to be repairedor replaced. Additionally, heavy maintenance results in unpredictabledowntime for individual aircraft due to the variable demand for repairedor replacement components and the time to repair, reproduce, and/orreplace the affected components.

SUMMARY

Systems and methods of the present disclosure include determining aperformance status of a selected component in an aircraft. Flight datamay be collected during a flight of the aircraft. Feature data isextracted from the flight data. The feature data relates to performanceof one or more components of the aircraft. An ensemble of relatedmachine learning models is applied to the feature data. Each model ischaracterized by a false positive rate and a false negative rate, andgenerally by a ROC (receiver operating characteristic).

Each model produces a positive score and a complementary negative scorerelated to performance of the selected component. The positive score ofeach model is weighted based on the false positive rate of the modelsuch that the weighted positive score is anti-correlated with the falsepositive rate. The weighted positive scores of the models of theensemble are combined, e.g., by summation, averaging, etc., to produce acombined weighted positive score. The negative score of each model isweighted based on the false negative weight of the model such that theweighted negative score is anti-correlated with the false negative rate.The weighted negative scores of the models of the ensemble are combined,e.g., by summation, averaging, etc., to produce a combined weightednegative score.

The performance status of the selected component is determined as one ofa positive category, a negative category, or an unclassified category.The positive category is determined if the combined weighted positivescore is greater than a threshold value and the combined weightednegative score is less than or equal to the threshold value. Thenegative category is determined if the combined weighted positive scoreis less than or equal to the threshold value and the combined weightednegative score is greater than the threshold value. The unclassifiedcategory is determined otherwise, i.e., if both of the combined weightedpositive score and the combined weighted negative score are greater thanthe threshold value or both are less than or equal to the thresholdvalue.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation of an aircraft and associatedpredictive maintenance system.

FIG. 2 is a schematic representation of an aircraft with subsystems ofthe aircraft.

FIG. 3 is a schematic illustration of an example of an aircraft.

FIG. 4 is a schematic representation of a predictive maintenance system.

FIG. 5 is an example of a receiver operating characteristic plot.

FIG. 6 is a schematic representation of predictive maintenance methods.

FIG. 7 is a schematic representation of a computerized system.

DESCRIPTION

Conventional approaches to servicing components that are not performingto specifications or that are at risk of non-performance includeproactive replacement at scheduled maintenance intervals (regardless ofperformance status) and testing (before, during, and/or after use). Forexample, many aircraft include built-in self-tests for individualcomponents that are performed before flight. Additionally oralternatively, the performance of a component may be monitored in use(e.g., as an active component is actuated). If the test indicates thecomponent is performing unexpectedly, the operator (e.g., aircrew)and/or service personnel are notified of the need for repair orreplacement. Generally, this type of testing provides only present, andpossibly past, indications of component performance and no indication offuture performance, such as impending non-performance. Proactivereplacement during scheduled maintenance replaces components based uponan estimate of the useful life of the population of components and doesnot rely on a prediction of when a particular component needsreplacement.

However, the history of operation of the aircraft and/or the individualcomponents provides insight into when a component may experience anon-performance event. Hence, the operational history and/or theoperational characteristics of the component may be utilized to reliablyschedule component maintenance prior to any unexpected performance. Asused herein, non-performing and non-performance include unexpectedperformance, degraded performance, and/or non-responsiveness.Non-performing and non-performance may include partial operation (e.g.,when a component performs expectedly in some situations but not others,or when a component provides a functional but inadequate response) andcomplete non-operation (e.g., when a component does not respond tocommands or other input, or when a component provides an unacceptableresponse).

As an example, aircraft may monitor their subsystems and overallperformance, and record system operation data, which may relate toactive and/or structural component health. For example, aircraft mayrecord speed, acceleration, flight time, number of take-offs andlandings, number of catapults, number of traps, etc. (and/or such datamay be recorded for individual aircraft). Some aircraft, such as F/A-18Hornet model aircraft, include accelerometers and/or strain gauges tomeasure overall aircraft motion and stresses applied to componentsand/or sections of the aircraft. However, simple measures such as totalflight hours, total number of traps, or peak acceleration do notreliably predict when or which components should be replaced.

For complex problems like predicting performance of aircraft components,machine learning algorithms may provide efficient solutions. However,even when guided and assisted by data scientists, many machine learningalgorithms may result in models that are class biased, i.e.,significantly better at predicting one outcome (class) over another.Models that have asymmetric prediction performance are referred to asclass-biased models. A class-biased model may have a low error rate fora single class, such as a ‘positive’ prediction, while having a higherror for a different class (or the other class in a two-class system).For example, a class-biased model may identify aircraft components inneed of service with great accuracy (e.g., a high true positive rate)but may also identify nearly all of the same type of components as inneed of service (whether service is actually needed or not). Becausesuch a class-biased model has a high error rate (e.g., a high falsepositive rate and a low true negative rate), the class-biased model doesnot sufficiently discriminate components (as needing service or not).

Systems and methods of the present disclosure may leverage the power ofclass-biased models (e.g., having a low false negative rate) whilelimiting the effect of the overall error of the class-biased models. Thedisclosed systems and methods utilize a group of models in an ensembleto leverage the strengths of all of the models to produce an aggregateresult that is more accurate than the underlying models and that doesnot suffer from the errors associated with class-biased models. Hence,systems and methods of the present disclosure do not need to rely on asingle model that has high accuracy (e.g., both high true positive andhigh true negative rates).

In the systems and methods of the present disclosure, each of the modelsof the ensemble is utilized to independently assess the input data. Theindependent assessments are aggregated by weighting the independentassessments according to the estimated performance of the respectivemodel (hence, weighting based on the trustworthiness or reliability ofthe individual models' prediction). Thus, unlike conventional weightingin which the weights are based on the number of models or consensusamong models, the systems and methods of the present disclosure createand/or apply weights that emphasize accurate models for a particularoutcome (class-biased models) when those models are likely to accuratelyassess the outcome.

For example, a given problem to assess component performance (e.g., inneed of repair or not) may yield five useful models: one relatively goodat identifying when repair is needed but also with a high false positiverate, one relatively good at identifying when a repair is not needed butalso with a high false negative rate, and three with in-between truepositive and false positive rates. The model that has a high falsepositive rate is weighted lower when it produces a positive case result.The model that has a high false negative rate is weighted lower when itproduces a negative case result. More generally, each of the models isweighted according to the false positive rate and false negative rate(and/or other measures of model performance and/or class bias) andaccording to the result of the respective model.

FIGS. 1-7 illustrate various aspects of predictive maintenance systemsand methods according to the present disclosure. In general, in thedrawings, elements that are likely to be included in a given embodimentare illustrated in solid lines, while elements that are optional oralternatives are illustrated in dashed lines. However, elements that areillustrated in solid lines are not essential to all embodiments of thepresent disclosure, and an element shown in solid lines may be omittedfrom a particular embodiment without departing from the scope of thepresent disclosure. Elements that serve a similar, or at leastsubstantially similar, purpose are labelled with numbers consistentamong the figures. Like numbers in each of the figures, and thecorresponding elements, may not be discussed in detail herein withreference to each of the figures. Similarly, all elements may not belabelled or shown in each of the figures, but reference numeralsassociated therewith may be used for consistency. Elements, components,and/or features that are discussed with reference to one or more of thefigures may be included in and/or used with any of the figures withoutdeparting from the scope of the present disclosure.

As illustrated in FIG. 1, a predictive maintenance system 10 includes acomputerized system 200 (as further discussed with respect to FIG. 7).The predictive maintenance system 10 may be programmed to perform,and/or may store instructions to perform, the methods described herein.

The predictive maintenance system 10 is associated with an aircraft 30.The predictive maintenance system 10 may be an independent system and/ormay be a part of the aircraft 30 (an on-board system, also referred toas an on-platform system), as schematically represented by theoverlapping boxes of aircraft 30 and predictive maintenance system 10 inFIG. 1. In particular, where the predictive maintenance system 10 isphysically independent of the aircraft 30, the predictive maintenancesystem 10 may be associated with a plurality of aircraft 30 (e.g., afleet of aircraft 30).

Aircraft 30 are vehicles configured for flight and include one or moresubsystems 32 that control, perform, and/or monitor one or more aspectsof aircraft operation. As indicated in FIG. 2, examples of subsystems32, which also may be referred to as systems, include an environmentalcontrol system 34, a propulsion system 36, a flight control system 38,an electrical system 40, and a hydraulic system 42. Subsystems 32 may bea portion of other systems or subsystems of aircraft 30. For example,subsystem 32 may be a rudder control system that is a subsystem of theflight control system 38. Subsystems 32 include one or more components44 that together may perform the function of the subsystem 32. Examplesof components 44 include an actuator, a servomechanism, an engine, amotor, an electronics module, a pump, a valve, and an airframe member.Components 44 may be referred to as parts, elements, units, etc., andmay be line replaceable and/or field replaceable (e.g., line replaceableunits). Components 44 may be subsystems of the respective subsystem 32.Components 44 may be active and/or controlled components, e.g.,components 44 may be configured to change state during flight of theaircraft 30. Components 44 may be electrical, optical, mechanical,hydraulic, fluidic, pneumatic, structural, and/or aerodynamiccomponents.

If a component 44 performs unexpectedly, or otherwise is non-performing,the operation of the corresponding subsystem 32 may be impacted. Manyaircraft 30 are designed to be tolerant of component non-performance.For example, aircraft 30 may include redundant subsystems 32 and/orsubsystems 32 may incorporate redundant components 44. Additionally oralternatively, subsystems 32 may be designed to safely operate with lessthan all components 44 fully operational, e.g., by operating withreduced performance.

As shown generally in FIGS. 1 and 2, subsystems 32 generally includesensors 46 configured to measure and/or monitor the performance ofindividual components 44, groups of components 44, and/or the subsystem32. Additionally or alternatively, sensors 46 may measure and/or monitorthe environmental condition, the condition of one or more components 44,and/or the inputs and/or outputs of the subsystem 32 and/or thecomponent(s) 44. Sensors 46 may be utilized in built-in testing,performance monitoring, and/or subsystem control. Sensors 46 may beand/or may include encoders, accelerometers, position sensors,electrical meters, electronic compasses, strain gauges, temperaturesensors, and/or pressure transducers.

Further, aircraft 30 may include a controller 50 that may be configuredto control and/or monitor one or more subsystems 32, components 44,and/or sensors 46. Controller 50 may be on board the aircraft 30 (alsoreferred to as on-platform), may be operated independent of the aircraft30, and may be a system independent of the aircraft 30 (also referred toas off-platform), as schematically represented by the overlapping boxesof aircraft 30 and controller 50 in FIG. 1. Controller 50 and predictivemaintenance system 10 may communicate via a data link 54. The data link54 is an electronic communication link and may include one or morewired, wireless, radio, optical, and/or electrical communicationchannels.

Generally, sensors 46 and/or controllers 50 are configured to collectdata during flight of the aircraft 30. The data collected from thesensors 46 are flight data 24. Data may include records of environmentalconditions (e.g., temperature, pressure, humidity), aircraft operation(e.g., airspeed, altitude, ground location, heading), flight performance(e.g., acceleration (in a direction and/or about an axis), verticalacceleration, strain, velocity (in a direction and/or about an axis),vertical velocity, pitch rate, roll rate, yaw rate, angle of attack,attitude, etc.), subsystem operation (actual operation), subsystemcommand status (expected operation), component operation (actualoperation), and/or component command status (expected operation).Controller 50 may be configured to store flight data 24, e.g., in aflight database 12, and may be referred to as a flight data storagesystem. The flight database 12 may be at least partially stored in thecontroller 50 and/or the predictive maintenance system 10. The flightdatabase 12 may include flight data 24 from more than one flight of theaircraft 30 and may include flight data 24 from a plurality of aircraft30 (e.g., a fleet of aircraft 30).

Actively-controlled components 44 may be controlled by signals from thecontroller 50 and/or a control input 48, optionally in one or morefeedback loops. Control inputs 48 are configured to accept operator(e.g., aircrew) inputs and to provide the operator inputs to thecontroller 50 and/or the component 44. Operator inputs may be aposition, a force, a pressure, and/or an angle applied to the controlinput 48. Examples of control inputs 48 include a control stick (alsocalled a yoke), an engine throttle lever, a trim switch, rudder pedals,and a flap control switch.

Generally, actively-controlled components 44 may be part of a feedbackloop in which a control signal from the controller 50 and/or the controlinput 48 commands the component 44 to react (e.g., move to a position)and in which one or more sensors 46 generate a response signal thatindicates the result of the command (e.g., a position achieved inresponse to the command to move). The controller 50 may process theresponse signal to generate a new (or modified) control signal to reducethe error between the intent of the command and the result of thecommand.

Components 44 that are active may be controlled based on static ordynamic set points, user inputs, and/or the state of other components44, the subsystem 32 that includes the component 44, other subsystems32, and/or the aircraft 30 (e.g., flight performance parameters such asacceleration (in a direction and/or about an axis), verticalacceleration, velocity (in a direction and/or about an axis), verticalvelocity, pitch rate, roll rate, yaw rate, angle of attack, attitude,etc.).

Feedback loops, where present, may incorporate feedback and controlsignals from one or more control inputs 48, one or more sensors 46,and/or settings of one or more other feedback loops. A feedback loop forone active component 44 generally incorporates signals from itsassociated control input(s) 48 and sensor(s) 46. The feedback loop mayincorporate signals from control inputs 48 for other components 44,sensors 46 associated with other components 44, and/or other sensors 46(supplied directly or indirectly through other controllers 50 or flightcomputers).

The predictive maintenance system 10 may include a display that isconfigured to present visual, audio, and/or tactile signals to anoperator and/or user of the predictive maintenance system 10. Thesignals may be configured to indicate system information, for example,indicating the identity of the selected component 44, a performancestatus and/or performance category (e.g., operational, good, degradedperformance, non-performing, impending non-performance, and/ormaintenance needed), that the selected component 44 is likely tonon-perform (or likely to perform), and/or the predicted remaininguseful life of the selected component 44 (e.g., the number of flightsbefore predicted non-performance). As further described with respect toFIG. 7, predictive maintenance system 10 and/or controller 50 mayinclude, and/or may be, a computerized system 200.

Where it is hard to predict when a component 44 may performunexpectedly, the urgency to repair a non-performing component may beheightened (e.g., to avoid having a subsystem 32 with more than onenon-performing component). Further, unexpected performance of somecomponents 44 may stress the respective subsystem 32 and may contributeto and/or cause other components 44 to perform unexpectedly. Hence, thenon-performing component 44 typically is immediately repaired whennon-performance is identified. The urgent repair may result in abortedflights and/or unscheduled maintenance of the aircraft, with theconsequent unscheduled downtime, and may strain the supply chain forreplacement parts. Unscheduled repair and downtime may lead to a loweravailability of the aircraft 30 and/or the lack of availability of theaircraft 30 at critical moments in time. As used herein, repair andreplacement of a component include installation of a different component(generally new or remanufactured) or repair of the original component.

Though examples may refer to aircraft 30, the systems and methods ofthis disclosure may be utilized with other apparatuses. For example,systems and methods of the present disclosure may be applied to othervehicles and/or machinery. Hence, a reference to aircraft 30 may bereplaced with a reference to a vehicle and/or machinery. Correspondingterms like flight may be replaced by terms like excursion and/oroperation; flying may be replaced by driving, operating, and/or running.

As used herein, the term ‘fleet’ refers to one or more of the subjectaircraft, vehicles, and/or machinery. A fleet may refer to all of thesubject aircraft, vehicles and/or machinery at a location, based at alocation, used for a similar purpose, and/or used by a corporate entity(e.g., a corporation, a military unit). For a fleet of aircraft, eachaircraft 30 may be substantially identical with the same types ofsubsystems 32 and the same types of components 44 in the subsystems 32.As used herein, components 44 of the same type are components 44 inequivalent locations, serving equivalent functions in the differentaircraft 30 and/or subsystems 32.

FIG. 3 illustrates an example of the aircraft 30 with various components44 that may be the subject of the systems and methods of the presentdisclosure. As illustrated in the example of FIG. 3, components 44generally are internal components such as actuators, servomechanisms,valves, airframe members, etc. Components 44 may be and/or may beassociated with external portions of the aircraft 30 such as flightcontrol surfaces, landing gear, etc.

The aircraft 30 may include various sensors 46 that may be associated(directly or indirectly) with a given component 44. Sensors 46 may beconfigured to measure environmental conditions (e.g., temperature,pressure, humidity), aircraft operation (e.g., airspeed, altitude,ground location, heading), flight performance (e.g., acceleration (in adirection and/or about an axis), vertical acceleration, strain, velocity(in a direction and/or about an axis), vertical velocity, pitch rate,roll rate, yaw rate, angle of attack, attitude, etc.), subsystemoperation (actual operation), subsystem command status (expectedoperation), component operation (actual operation), and/or componentcommand status (expected operation).

The aircraft 30 includes one or more control inputs 48 to providecontrol of the aircraft 30 and components 44 of the aircraft 30. Thecontrol inputs 48 typically are located in the cockpit and/or areaccessible by the aircrew.

FIG. 4 is a schematic representation of a predictive maintenance system10. The predictive maintenance system 10 is configured to utilizefeature data 26 extracted from flight data 24 to produce a performanceindicator 28 that reflects the performance status of the component 44(e.g., current condition or the condition within a given number offuture flights). For example, the performance indicator 28 may be thefuture performance likelihood, the likelihood of non-performance,whether non-performance is imminent or not, or whether the component 44is performing as expected or not. The predictive maintenance system 10may be configured to utilize flight data 24 along with feature data 26to produce the performance indicator 28.

The performance indicator 28 also may be referred to as the category,prediction, and/or estimate of performance of the component 44 duringfuture flights (e.g., an estimate of the likelihood of non-performancealso may be referred to as a prediction of future non-performance). Theflight data 24 and/or extracted feature data 26 may relate to thecomponent 44 directly or may relate to the associated subsystem 32and/or aircraft 30. The component 44 that is the subject of thepredictive maintenance system 10 may be referred to as the subjectcomponent and/or the selected component.

The predictive maintenance system 10 may be part of a health managementsystem and/or a health assessment system for the associated aircraft 30(on-platform or off-platform). Additionally or alternatively, thepredictive maintenance system 10 may be utilized to create and/or deploypredictive models for a health management system and/or a healthassessment system. The health management system and/or the healthassessment system may be configured to monitor, assess, and/or indicatethe operational status of one or more components 44 of the aircraft 30.

As schematically represented in FIG. 4, the predictive maintenancesystem 10 includes several modules (e.g., instructions and/or dataconfigured to be executed by a computerized system as described withrespect to FIG. 7). These modules may be referred to as agents,programs, processes, and/or procedures.

The predictive maintenance system 10 includes a performanceclassification module 60. The performance classification module 60includes an ensemble 64 of related primary models 66 and an aggregationmodule 68. The performance classification module 60 is configured toclassify extracted feature data 26, i.e., to utilize feature data 26extracted from flight data 24 to form the performance indicator 28 ofthe performance status of the component 44 (e.g., the futureperformance, the likelihood of non-performance, whether anon-performance event is imminent or not, whether the component 44 isperforming as expected or not). The performance classification module 60may be configured to utilize flight data 24 along with feature data 26to produce the performance indicator 28. The predictive maintenancesystem 10 and/or the performance classification module 60 may provide aremaining useful life (RUL) estimate of the number of flights beforeactual non-performance is expected to occur.

Each of the primary models 66 (also referred to as models and machinelearning models) of the ensemble 64 are machine learning algorithms thatidentify (e.g., classify) the category (one of at least two states) towhich a new observation (set of extracted features) belongs. As usedherein, the adjective “primary,” when modifying model, classifier,category, and related elements, indicates a principal, base,fundamental, and/or limited element. Hence, the primary models 66 of theensemble 64 form a basis for the performance classification module 60 toproduce the performance indicator 28.

Primary models 66 are selected and/or configured to transform extractedfeature data 26 into an estimate of performance status by producing apositive classification score 76. The positive classification score 76,which may be referred to as the positive score, is a numeric value thatrelates to the positive outcome or result of the primary model 66.Primary models 66 also produce a negative classification score 78 thatis complementary to the positive classification score 76. The negativeclassification score 78, which may be referred to as a negative score,is a numeric value that relates to the negative outcome or result of theprimary model 66.

The negative classification score 78 and the positive classification 76score are complementary in that, for a two-class classification system,the combination of the negative classification score 78 and the positiveclassification score 76 encompass the entire range of possible outcomesfor the primary model 66 (e.g., the performance status). For example,the positive classification score 76 and the negative classificationscore 78 may be likelihood estimates of respective positive and negativeoutcomes, and the positive classification score 76 and the negativeclassification score 78 may sum to unity (100%). Because the positiveclassification score 76 and the negative classification score 78 arecomplementary, one of the classification scores may be derived from theother. In some embodiments, the primary model 66 directly produces onlyone of the positive classification score 76 and the negativeclassification score 78. The other classification score may be derivedfrom the classification score produced by the primary model 66.

The classification scores may be binary outcomes (e.g., performanceacceptable or non-performance expected), likelihoods of non-performanceor expected performance, etc. that reflect the performance status of thecomponent 44 (e.g., current performance or expected performance for thegiven number of future flights). For example, the positiveclassification score 76 may be an estimate of the likelihood of theselected component's non-performance or need of repair and the negativeclassification score 78 may be an estimate of the likelihood of theselected component's performance or need of repair.

Primary models 66 of the ensemble 64 are related in that each of theprimary models 66 relate to the same component 44 and to the same typeof positive and negative classes (e.g., the same performance status).However, the primary models 66 are different machine learning models(e.g., using different algorithms or different parameters). The ensemble64 may include two or more primary models 66. The positive and negativeclasses may be likelihoods of non-performance and expected performance,likelihoods of a non-performance event or no non-performance event, etc.Classes relating to continued performance may be referred to as “noimpending non-performance,” “no impending non-performance event,” “nomaintenance needed,” “good,” and/or “low risk.” Classes relating tonon-performance may be referred to as “impending non-performance,”“impending non-performance event,” “maintenance needed,” and/or “highrisk.”

The terms positive and negative with respect to primary model 66outcomes, predictions, scores, classes, cases, and/or performance referto complementary states associated with the primary models 66 and do notgenerally refer to beneficial or detrimental states of the primary model66 and/or the selected component 44. In a two-class classificationsystem, positive refers to one of the classes and negative refers to theother class. Generally, primary models 66 relate to two classes (e.g.,positive-negative, true-false, yes-no, binary values, etc.). Primarymodels 66 that relate to more than two classes may be mapped to one ormore models that relate to two classes. For example, three classes,e.g., A, B, and C, may be mapped to three different two-classclassifications, i.e., A or not A, B or not B, and C or not C.

Each primary model 66 may be characterized by performance with trainingor verification inputs (e.g., using inputs with known results). Themodel outcomes may be categorized as true positive outcomes (thepredicted value of the model is positive and the known result is alsopositive), true negative outcomes (the predicted value is negative andthe known result is negative), false positive outcomes (the predictedvalue is positive and the known result is negative), or false negativeoutcomes (the predicted value is negative and the known result ispositive). True positives and true negatives are correct predictions ofthe primary model 66. False positives and false negatives are incorrectpredictions of the primary model 66. A false positive is also called atype I error. A false negative is also called a type II error.

True positive rate, also called the sensitivity and/or the recall, isthe total number of true positives divided by the total number of knownpositive results. Positive predictive value, also called the precision,is the total number of true positives divided by the total number ofpredicted positive values. True negative rate, also called thespecificity, is the total number of true negatives divided by the totalnumber of known negative results. Negative predictive value is the totalnumber of true negatives divided by the total number of predictednegative values. False positive rate, also called the fall-out, is thetotal number of false positives divided by the total number of knownnegative results. False discovery rate is the total number of falsepositives divided by the total number of predicted positive values.False negative rate is the total number of false negatives divided bythe total number of known positive results. False omission rate is thetotal number of false negatives divided by the total number of predictednegative values. Accuracy is the total number of true positives and truenegatives divided by the total population. Unless more than two statesare possible, the true positive rate is the complement of the falsenegative rate (i.e., the true positive rate and the false negative rateadd to unity) and the true negative rate is the complement of the falsepositive rate (i.e., the true negative rate and the false positive rateadd to unity).

Each primary model 66 may be characterized by its performance, forexample by a true positive rate, a false positive rate, a true negativerate, and/or a false negative rate. The performance may be characterizedby a performance metric that incorporates a cost component and a benefitcomponent that relate to the performance of the primary model 66. Forexample, the primary models 66 may each be characterized by a receiveroperating characteristic (ROC), which is an order pair of the falsepositive rate (cost) and the true positive rate (benefit). Other typesof cost and benefit measures (e.g., false negative rate, true negativerate, positive predictive value, negative predictive value, falsediscovery rate, false omission rate, and/or accuracy) may be utilized ina similar manner. Generally, the primary models 66 of the ensemble 64have different performance characteristics. Each of the primary models66 may have a unique performance (e.g., ROC).

FIG. 5 is a schematic plot of ROC values of a group of candidate models.Each candidate model is characterized by an ROC performance point 82.Perfect performance would be when the true positive rate is unity andthe false positive rate is zero (upper left corner of the plot).Completely imperfect performance would be when the true positive rate iszero and the false positive rate is unity (lower right corner of theplot). Better performing models are more ‘northwest’ in the plot (towardthe upper left corner). The set of models that outperform all othermodels for each true positive and false positive rate form a convex hull84 (also referred to as a convex envelope). In the example of FIG. 5,these ‘best’ performing models are indicated at 86 a, 86 b, 86 c, and 86d. The convex hull 84 is a curve that extends from the point at 0 falsepositive rate and 0 true positive rate to the point at 1 false positiverate and 1 true positive rate. Between the endpoints, the convex hull 84is formed of line segments that connect ROC performance points 82 suchthat the performances of all candidate models are at or below (towardscompletely imperfect performance) the convex hull 84. The set of modelsthat form the convex hull 84 (the models indicated at 86 a-86 d) performbetter than other candidate models and may be referred to as best and/oroptimal among the group of candidate models.

Primary models 66 of the ensemble 64 may be selected from a larger groupof models (as illustrated in FIG. 5) and may be selected to be along (orsubstantially along) the convex hull 84. For example, the primary models66 of the ensemble 64 may be all, or a subset, of the models along theconvex hull 84. Primary models 66 of the ensemble 64 may be dispersedalong convex hull 84, i.e., generally spaced apart and definingessentially the entire convex hull 84. Primary models 66 of the ensemble64 may be uniformly distributed along the convex hull 84, i.e., spacedapart along the convex hull 84 with a substantially uniform distancebetween neighboring points along the convex hull 84.

The ensemble 64 generally includes at least two models with distinctlydifferent character, for example, one model that is class biased topositive outcomes and one model that is class biased to negativeoutcomes. For example, the ensemble 64 formed from models illustrated inFIG. 5 may include the model 86 a , which has a low false positive rateand a low true positive rate, and the model 86 d , which has a highfalse positive rate and a high true positive rate. The ensemble 64 mayinclude the model along the convex hull 84 that has the lowest falsepositive rate coupled with a non-trivial (i.e., non-zero) true positiverate, for example the model 86 a . The ensemble 64 may include the modelalong the convex hull 84 that has the highest true positive rate coupledwith a non-trivial (i.e., non-unity) false positive rate, for examplethe model 86 d . The models 86 b and 86 c along the convex hull 84 haveintermediate false positive rates and true positive rates relative tothe models 86 a and 86 d .

Candidate models and primary models 66 may be the result of supervisedmachine learning and/or guided machine learning in which training data(extracted feature data 26 which correspond to known outcomes) areanalyzed to discern the underlying functional relationship between theextracted feature data 26 and the outcomes (the positive classificationscore 76 and the negative classification score 78). The underlyingfunction may be an analytical function, a statistical correlation (e.g.,a regression), and/or a classification algorithm. Examples ofstatistical correlations include logistic regression and probitregression. Examples of classification algorithms include naive Bayesclassifiers, support vector machines, learned decision trees, ensemblesof learned decision trees (e.g., random forests of learned decisiontrees), and neural networks. Models that employ classificationalgorithms may be referred to as classifiers. Generally, supervisedmachine learning includes determining the input feature data 26 (thefeatures extracted from the underlying training data), determining thestructure of the learned function and corresponding learning algorithm(e.g., support vector machine and/or learned decision trees), applyingthe learning algorithm to the training data to train the learnedfunction (e.g., the primary model 66), and evaluating the accuracy ofthe learned function (e.g., applying the learned function to a test dataset with known outcomes to verify the performance of the learnedfunction).

Returning to FIG. 4, the performance classification module 64 includesthe aggregation module 68. The aggregation module 68 transforms theinput positive classification scores 76 and the negative classificationscores 78 from all of the primary models 66 of the ensemble 64 of modelsinto the performance indicator 28. The performance indicator 28 reflectsthe performance status of the component (e.g., present performance orperformance within the given number of future flights). The performanceindicator 28 may be called the aggregate indicator and/or the aggregatecategory.

The aggregation module 68 weights the positive classification scores 76and the negative classification scores 78 according to the individualperformances of the primary models 66. Primary models 66 that morereliably predict positive outcomes are weighted higher when that modelproduces a positive outcome (e.g., positive classification score 76indicative of the positive class). Primary models 66 that more reliablypredict negative outcomes are weighted higher when that model produces anegative outcome (e.g., negative classification score 78 indicative ofthe negative class). Similarly, primary models 66 that unreliablypredict a particular outcome (positive or negative) are weighted lowerwhen that model produces the particular outcome. Hence, the aggregationmodule 68 may compensate for the inclusion of class-biased models withinthe ensemble 64. The combined, weighted outcomes of the primary models66 may more reliably indicate the performance status of the component 44than any one of the primary models 66 (or any un-weighted combination ofthe primary models 66).

The weighting scheme performed by the aggregation module 68 may bereferred to a trust weighting and/or reliability weighting. The positiveclassification scores 76 and/or the negative classification scores 78may be weighted to emphasize and/or deemphasize primary models 66according to the classification score (or scores). For each positiveclassification score 76 of the primary models 66, the weight that may beapplied may be anti-correlated with the false positive rate (or othermeasure of the unreliability of a ‘positive’ determination). For eachnegative classification score 78 of the primary models 66, the weightthat may be applied may be anti-correlated with the false negative rate(or other measure of the unreliability of a ‘negative’ determination).The weights are anti-correlated with the respective measures in that theweight increases as the measure decreases and the weight decreases asthe measure increases. Thus, the primary models 66 with low falsepositive rates may be treated as more trustworthy and/or reliable whenthey predict a positive outcome. Likewise, the primary models 66 withlow false negative rates may be treated as more trustworthy and/orreliable when they predict a negative outcome.

The same anti-correlated weighting function (a positive weightingfunction) may be applied to each of the positive classification scores(adjusted for the specific false positive rate of each primary model66). The same anti-correlated weighting function (a negative weightingfunction) may be applied to each of the negative classification scores(adjusted for the specific false negative rate of each primary model66). Typically, the positive weighting function and the negativeweighting function are different because the primary models 66 (and moregenerally the candidate models used to select the primary models 66) areasymmetric in performance. For example, it may be more difficult toproduce a model with an incrementally better true positive rate than amodel with an incrementally better true negative rate (or vice versa).

The positive weighting function and the negative weighting function maytake the form of an inverse exponential function with an exponentproportional to the respective false positive rate (positive weightingfunction) or the false negative rate (negative weighting function). Forexample, the weighted positive score (the product of the positiveweighting function and the positive classification score of a primarymodel) may be given by:

W _(p) =Pe ^(−αFPR)

where W_(p) is the weighted positive score, P is the positiveclassification score, FPR is the false positive rate, and α is aconstant. The constant α may be determined according to the particularperformances of the primary models 66 by optimizing the accuracy of theweighted ensemble 64 with a given set of training data (in which theperformance outcomes are known). The weighted negative score (theproduct of the negative weighting function and the negativeclassification score of a primary model) may be given by:

W _(N)=Ne^(−FNR)

where W_(N) is the weighted negative score, N is the negativeclassification score, FNR is the false negative rate, and ß is aconstant. The constant ß may be determined according to the particularperformances of the primary models 66 by optimizing the accuracy of theweighted ensemble 64 with a given set of training data (in which theperformance outcomes are known).

The weights may be normalized and/or scaled. As examples, the positiveweights may be normalized such that the individual positive weightingfunctions for each primary model 66 sum to unity. The positive weightsmay be scaled by dividing the individual positive weighting functionsfor each primary model 66 by the total number of primary models 66.Likewise, the negative weights may be normalized or scaled in the samemanner.

The weighted positive scores of the primary models 66 may be combined bysumming (and/or averaging, etc.) the individual weighted positive scoresfor all (or a subset, e.g., a filtered subset) of the primary models 66.If a subset of primary models 66 contribute to the combined weightedpositive score, the subset may be determined according to the positiveclassification score 76, the negative classification score 78, theweighted positive score, and/or the weighted negative score. Forexample, only primary models 66 with a positive classification score 76over a predetermined threshold may contribute to the combined weightedpositive score. The weighted negative scores of the primary models 66may be combined in a manner analogous to the weighted positive scores.

The performance indicator 28 may be determined according to the combinedweighted positive score and/or the combined weighted negative score. Thecombined weighted positive score and/or the combined weighted negativescore may be compared to one or more thresholds to determine theperformance indicator 28. For example, one threshold may distinguish apositive category of the performance indicator 28 from a negativecategory. As another category, the positive category may be assigned tovalues greater than a first threshold and the negative category may beassigned to values less than a second threshold. Values between thefirst and second threshold may be assigned an unclassified orundetermined category. Further, different thresholds may be applied todifferent ones of the combined weighted positive score and the combinedweighted negative score to determine the performance indicator 28 basedon both of the combined weighted positive score and the combinedweighted negative score. The thresholds may be determined by optimizingthe accuracy of the performance indicator 28 with a given set oftraining data (in which the performance outcomes are known).

As a specific method of determining the performance indicator 28, theperformance indicator 28 may be determined to be a positive category ifthe combined weighted positive score is greater than a threshold or anegative category if the combined weighted positive score is less thanthe same threshold. As another method, the performance indicator 28 maybe determined to be a positive category if the combined weightedpositive score is greater than a first threshold or a negative categoryif the combined weighted negative score is greater than a secondthreshold. As yet another method, the performance indicator 28 may bedetermined to be:

a positive category if the combined weighted positive score is greaterthan a first threshold and the combined weighted negative score is lessthan or equal to a second threshold;

a negative category if the combined weighted positive score is less thanor equal to the first threshold and the combined weighted negative scoreis greater than the second threshold; or

an unclassified category (or an undetermined category) otherwise (i.e.,if both combined weighted scores are greater than the respectivethreshold or both are less than or equal to the respective threshold).

The first and second thresholds of these examples may be the same. Wherethe thresholds of these examples are compared to other values, thecomparison may be equal, less than, less than or equal to, greater than,or greater than or equal to. Such comparison may be substituted for oneanother without departing from the scope of this disclosure.

Further, the performance indicator 28 may reflect a confidence level ofthe determined category. The confidence level may be determined byranking the combined weighted positive score and/or the combinedweighted negative score for the same component 44 on different aircraft30. Components 44 ranking at the top of the corresponding componentsmerit a higher confidence level that components 44 ranking at the bottomof the corresponding components 44. Additionally or alternatively, theconfidence level may be determined by comparing the combined weightedpositive score and/or the combined weighted negative score to thethreshold(s) utilized to determine the category of the performanceindicator 28. Scores near the respective threshold merit a lowerconfidence level than scores significantly different from the respectivethreshold.

Where used to predict the future performance of the component 44, thegiven number of future flights may be different for each of the primarymodels 66 (though in some embodiments, the given number of futureflights may be the same for each of the primary models 66 and theperformance indicator 28). The performance indicator 28 may reflectperformance within a threshold number of future flights that is relatedto the given numbers of future flights associated with the primarymodels 66 (e.g., the threshold number may be the maximum of the givennumbers). The number of future flights generally is a time window orhorizon that is a relevant period of time to establish the performancestatus of the component 44 (by the primary models 66, the aggregationmodule 68, and/or the performance indicator 28). The classificationscores of the primary models 66 and the performance indicator 28 mayindicate that a non-performance event will or will not happen sometimeduring the period. Additionally or alternatively, the classificationscores of the primary models 66 and the performance indicator 28 mayindicate a time until a non-performance event, if one is predicted tooccur within the given number of future flights.

The given or threshold number of future flights may be 1 flight, 2flights, 3 flights, 5 flights, 10 flights, or more. The primary models66 of the ensemble 64 may relate to a range of given numbers. Forexample the ensemble 64 may include a first primary model 66 with afuture flight window of 1 flight, a second primary model 66 with afuture flight window of 2 flights, and up to an Nth primary model 66with a future flight window of N flights. An ensemble 64 may relate to afuture flight window of k flights by including primary models 66 thatrelate to various subsets of the k flight window (e.g., a sequentialseries of windows of 1 flight, 2 flights, 3 flights, . . . , k flights).The aggregation of the various smaller flight windows of the primarymodels 66 combined to predict the performance status within the largerflight window (i.e., k flights) of the ensemble 64 as a whole (and theperformance indicator 28) may increase the accuracy of the aggregateprediction (the performance indicator 28) relative to the individualpredictions (i.e., the positive classification scores 76 and/or thenegative classification scores 78) of the primary models 66.

Predictive maintenance systems 10 may include a feature extractionmodule 62 that is configured to extract feature data 26 from flight data24 collected during a flight of the aircraft 30. The flight data 24 andthe extracted feature data 26 may relate to the performance of theaircraft 30 (e.g., flight performance), the subsystem that includes theselected component, and/or the selected component. The flight data 24may be collected during a single flight or a series of flights. Usingflight data 24 from a series of flights may provide a more reliableprediction of component performance because of the greater amount offlight data 24 and/or because the aircraft 30, the subsystem 32, and/orthe component 44 are more likely to be subjected to a greater range ofconditions and/or particular stress conditions.

Examples of flight data 24 include an indication of weight on wheels,sensor status (e.g., operating normally, degraded performance,non-responsive), subsystem settings (e.g., propulsion output, payloadpresence, etc.), component settings (e.g., flight control surface isdeployed), sensor values (e.g., airspeed, acceleration, verticalacceleration, strain, velocity, vertical velocity, pitch rate, rollrate, yaw rate, angle of attack, attitude, altitude, heading, position,etc.), control input values (e.g., a control stick position, a controlstick lateral position, a control stick longitudinal position, a rudderpedal position, a rudder pedal differential position), engine throttle,a voltage, a current, ambient temperature, and ambient pressure.

Flight data 24 may be collected systematically, e.g., consistently onsubstantially every flight, consistently in substantially every aircraft30, and/or on a consistent basis (e.g., periodically). Flight data 24relating to different sensors 46 and/or control inputs 48 may becollected at different times or at substantially the same time. Flightdata 24 relating to the same sensor 46 or control input 48 generallyforms a time series (e.g., periodic, quasi-periodic, or aperiodic).

The feature extraction module 62 may be configured to extract featuredata 26 that may be correlated to and/or may indicate likely componentperformance. The feature extraction module 62 may be configured todetermine a statistic of sensor values and/or control input valuesduring a time window, a difference of sensor values and/or control inputvalues during a time window, a difference between sensor values and/orcontrol input values measured at different locations and/or differentpoints in time, and/or a statistic of derived sensor values and/orcontrol input values (e.g., an average difference, a moving averageetc.). Such differences and/or statistics may be referred to as featuredata and/or extracted feature data. Feature data generally is derivedfrom sensor values and/or control input values that relate to the samesensed parameter (e.g., a pressure, a temperature, a speed, a voltage,and a current) and/or the same component 44. The statistic of values mayinclude, and/or may be, a minimum, a maximum, an average, a movingaverage, a variance, a deviation, a cumulative value, a rate of change,and/or an average rate of change. Additionally or alternatively, thestatistic of values may include, and/or may be, a total number of datapoints, a maximum number of sequential data points, a minimum number ofsequential data points, an average number of sequential data points, anaggregate time, a maximum time, a minimum time, and/or an average timethat the values are above, below, or about equal to a threshold value.The time window may be the duration of a flight of the aircraft, aportion of the duration of a flight of the aircraft, or a period of timeincluding one or more flights of the aircraft. For example, the timewindow may be at least 0.1 seconds, at least 1 second, at least 10seconds, at most 100 seconds, at most 10 seconds, at most 1 second,about 1 second, and/or about 10 seconds.

Additionally or alternatively, the feature extraction module 62 may beconfigured to analyze and/or extract sensor values and/or control inputvalues within certain constraints. For example, sensor values and/orcontrol input values may be subject to analysis only if within apredetermined range (e.g., outlier data may be excluded) and/or if othersensor values and/or control input values are within a predeterminedrange (e.g., one sensor value may qualify the acceptance of anothersensor value).

The ensemble 64 of related primary models 66 may be applied to thefeature data 26 extracted from the flight data 24 from one or moreprevious flights. Generally, each primary model 66 is supplied the samefeature data 26.

Predictive maintenance systems 10 may include a flight data collectionmodule 70 that is configured to collect flight data 24 during one ormore flights. Flight data 24 may be collected by the controller(s) 50and/or the sensor(s) 46 as described herein. The flight data collectionmodule 70 may be configured to collect flight data 24 automatically,e.g., whenever the aircraft 30 is flown.

Predictive maintenance systems 10 may include a display module 72 thatis configured to communicate the results of the performanceclassification module 60 to the user (e.g., the aircrew or ground crew).For example, the display module 72 may be configured to communicate theperformance indicator 28. The display module 72 may be configured tocommunicate system information, such as the identity of the selectedcomponent 44.

FIG. 6 schematically illustrates predictive maintenance methods 100 fora selected component (such as component 44). Methods 100 may be utilizedto impact health management of aircraft systems. By reliably predictingfuture component performance (e.g., a future non-performance event) andthereby enabling a more predictive schedule for repairs and demand forspare parts, methods 100 may contribute to overall aircraft maintenance,fleet management, and material logistics. Methods 100 may includemethods of preventative maintenance, methods of determining performancestatus, and/or methods of determining impending non-performance ofcomponents and/or systems.

Methods 100 include calculating performance 108 of the selectedcomponent based on feature data (e.g., feature data 26) and methods 100may include extracting 104 feature data (e.g., feature data 26) fromflight data (e.g., flight data 24) and/or determining 110 performancestatus of the selected component. Methods 100 may include operatingand/or utilizing the predictive maintenance system 10.

Calculating performance 108 includes calculating a performance indicator(e.g., performance indicator 28) that reflects a performance status ofthe selected component of the aircraft (e.g., aircraft 30). For example,the performance indicator may reflect whether the selected component islikely to perform or not within the given number of future flights.Calculating performance 108 may include operating and/or utilizing theperformance classification module 60.

Calculating performance 108 may include scoring models 120, weightingmodels 122, and aggregating models 124. Scoring models 120 includesapplying an ensemble of related models (e.g., ensemble 64 of relatedprimary models 66) to the feature data extracted from flight data toproduce a positive classification score and/or a negative classificationscore (e.g., positive classification score 76 and/or negativeclassification score 78). As described with respect to the ensemble 64and the primary models 66, the models may be characterized by variousperformance measures such as the ROC, false positive rate, true positiverate, etc. The ensemble of related models may include models withdifferent ROC and may include models from a convex hull (i.e., convexhull 84).

Weighting models 122 includes weighting the positive classificationscore and/or the negative classification score of each model of theensemble according to the individual performance measures of the modelsto respectively produce a weighted positive score and/or a weightednegative score. Weighting models 122 includes trust weighting, asdescribed with respect to ensemble 64, primary models 66, andaggregation module 68. Models that more reliably predict positiveoutcomes are weighted higher when that model produces a positiveoutcome. Models that more reliably predict negative outcomes areweighted higher when that model produces a negative outcome. Weightsapplied to the positive classification scores and/or the negativeclassification scores may be anti-correlated with the respective falsepositive rate (or other measure of the unreliability of a ‘positive’determination) and the false negative rate (or other measure of theunreliability of a ‘negative’ determination). Weighting models 122 mayinclude operating and/or utilizing the aggregation module 68.

Aggregating models 124 includes combining the weighted positive and/ornegative scores and determining the performance indicator according tothe combined weighted scores. Aggregating models 124 may include summing(and/or averaging, etc.) the individual weighted positive scores and/orsumming (and/or averaging, etc.) the individual weighted negativescores, as described with respect to ensemble 64, primary models 66, andaggregation module 68. The performance indicator may be determined basedon the combined weighted positive score and/or the combined weightednegative score relative to one or more thresholds as described withrespect to ensemble 64, primary models 66, and aggregation module 68.Aggregating models 124 may include operating and/or utilizing theaggregation module 68.

Methods 100 may include extracting 104 feature data from flight datacollected during a flight of the aircraft. As described herein, flightdata and/or feature data may relate to the performance of the aircraft(e.g., flight performance), a subsystem of the aircraft that includesthe selected component, and/or the selected component. Extracting 104may include operating and/or utilizing the feature extraction module 62.Extracting 104 may include determining a statistic of sensor valuesduring a time window, a difference of sensor values during a timewindow, and/or a difference between sensor values measured at differentlocations and/or different points in time as described with respect tothe feature extraction module 62.

Methods 100 may include collecting 102 flight data during a flight ofthe aircraft. Collecting 102 may include collecting flight data for aseries of flights. Collecting 102 may include operating and/or utilizingthe flight data collection module 70, the controller(s) 50, thesensor(s) 46, the control input(s) 48, and/or the aircraft 30 (e.g., ifflight data collection module 70 is configured to collect flight datawhenever the aircraft is flown). Methods 100 may include flying 114 theaircraft. Flying 114 the aircraft may cause collecting 102. Flying 114may include routine flying or flying to stress and/or to test theaircraft, the subsystem including the selected component, and/or theselected component.

Methods 100 may include displaying the performance status and/or theperformance indicator (and/or a representation relating to theperformance status and/or the performance indicator) by visual, audio,and/or tactile display, for example, by utilizing and/or operating theinput-output device 216 and/or the display module 72. Additionally oralternatively, methods 100 may include signaling by visual, audio,and/or tactile indicator that the selected component is likely toperform or to have a non-performance event within the threshold numberof future flights. The displaying and/or signaling may include on-board(on platform) and/or off-board (off platform) displays and/or signals.

Methods 100 may include determining 110 the performance status of theselected component based on the performance indicator. Determining 110may include determining whether the selected component is likely toperform or not, presently and/or within the given number of futureflights. Determining 110 may include determining the state of theperformance indicator and/or evaluating the value of the performanceindicator relative to a predetermined limit (e.g., less than, greaterthan, and/or about equal to the limit). For example, the need formaintenance may be associated with performance indicators indicating animpending-non-performance state with a likelihood greater than apredetermined limit.

Methods 100 may further include repairing 112 the selected component.Repairing 112 may include repairing, replacing, refurbishing,mitigating, and/or servicing (e.g., lubricating, cleaning) the selectedcomponent. Methods 100 may include determining whether to repair and/orrepairing 112 upon determining 110 the performance status (e.g.,determining that a repair would be useful and/or warranted based on theperformance indicator). For example, determining whether to repair mayinclude evaluating the value of the performance indicator relative to apredetermined limit (e.g., less than, greater than, and/or about equalto the limit). Where methods 100 include a form of repairing (e.g.,repairing 112, repairing 112 upon determining that a repair would beuseful, and/or determining whether to repair), methods 100 may bereferred to as preventative maintenance methods.

FIG. 7 schematically represents a computerized system 200 that may beused to implement and/or instantiate the methods, components, andfeatures described herein, for example, predictive maintenance systems10 and associated components, such as controller 50, flight datacollection module 70, feature extraction module 62, performanceclassification module 60, aggregation module 68, and/or display module72. The computerized system 200 includes a processing unit 202operatively coupled to a computer-readable memory 206 by acommunications infrastructure 210. The processing unit 202 may includeone or more computer processors 204 and may include a distributed groupof computer processors 204. The processing unit 202 may include, or beimplemented on, programmable, reconfigurable, and/or dedicated hardwaresuch as field-programmable gate arrays, digital signal processors,and/or application specific integrated circuits.

The computerized system 200 also may include a computer-readable storagemedia assemblage 212 that is operatively coupled to the processing unit202 and/or the computer-readable memory 206, e.g., by communicationsinfrastructure 210. The computer-readable storage media assemblage 212may include one or more non-transitory computer-readable storage media214 and may include a distributed group of non-transitorycomputer-readable storage media 214. The computer-readable memory 206,the computer-readable storage media assemblage 212, and thenon-transitory computer-readable media 214 are each computer readablemedia. Computer-readable media are tangible and are not merelytransitory signals.

The communications infrastructure 210 may include a local data bus, acommunication interface, and/or a network interface. The communicationsinfrastructure 210 may be configured to transmit and/or to receivesignals, such as electrical, electromagnetic, optical, and/or acousticsignals. For example, the communications infrastructure 210 may beconfigured to manage data link 54.

The computerized system 200 may include one or more input-output devices216 operatively coupled to the processing unit 202, thecomputer-readable memory 206, and/or the computer-readable storage mediaassemblage 212. Input-output devices 216 may be configured for visual,audio, and/or tactile input and/or output from and/or to the operator(e.g., the aircrew). Each input-output device 216 independently may beconfigured for only input, only output, primarily input, primarilyoutput, and/or a combination of input and output. Examples ofinput-output devices 216 include monitors (e.g., video monitor),displays (e.g., alphanumeric displays, lamps, and/or LEDs), keyboards,pointing devices (e.g., mice), touch screens, speakers, buzzers, andcontrols (e.g., buttons, knobs, etc.). The display module 72 may includeand/or may be an input-output device 216.

The computerized system 200 may include a distributed group ofcomputers, servers, workstations, etc., which each may be interconnecteddirectly or indirectly (including by network connection). Thus, thecomputerized system 200 may include one or more processing units 202,computer-readable memories 206, computer-readable storage mediaassemblages 212, and/or input-output devices 216 that are locatedremotely from one another.

One or both of the computer-readable memory 206 and thecomputer-readable storage media assemblage 212 include control logic 220and/or data 222. Control logic 220 (which may also be referred to assoftware, firmware, gateware, and/or hardware) may include instructionsthat, when executed by the processing unit 202, cause the computerizedsystem 200 to perform one or more of the methods described herein.Control logic 220 and/or data 222 may include applications (e.g., acontrol application), resources, access controls, and/or associatedinformation. Control logic 220 may include one or more of the flightdata collection module 70, feature extraction module 62, performanceclassification module 60, ensemble 64 of related primary models 66,primary models 66, aggregation module 68, and/or display module 72. Data222 may include flight database 12, flight data 24, feature data 26,and/or data associated with the modules, models, and/or methodsdescribed herein.

Where modules, models, and/or methods are described as performing one ormore functions, the respective device is configured, e.g., programmed,to perform the function(s). The respective device may include one ormore programs, agents, services, and/or components configured, e.g.,programmed, to perform the function(s) when the programs, agents,services, and/or components are executed by the processing unit 202 orotherwise operated by the computerized system 200. The control logic 220and/or data 222 may include instructions and/or informationcorresponding to the programs, agents, services, and/or components.

Examples of inventive subject matter according to the present disclosureare described in the following enumerated paragraphs.

A1. A method of determining a performance status of a selected componentin an aircraft, the method comprising:

extracting feature data from flight data collected during a flight ofthe aircraft, wherein the feature data relates to performance of one ormore components of the aircraft;

applying an ensemble of related models to produce, for each model, apositive score and a complementary negative score related to performanceof the selected component, optionally within a given number of futureflights of the aircraft, wherein each model is characterized by a falsepositive rate and a false negative rate, and optionally has an ROC(receiver operating characteristic);

weighting the positive score of each model to produce a weightedpositive score for each model based on the false positive rate of thatmodel such that the weighted positive score of each model isanti-correlated with the false positive rate of that model;

weighting the negative score of each model to produce a weightednegative score for each model based on the false negative rate of thatmodel such that the weighted negative score of each model isanti-correlated with the false negative rate of that model; and

determining the performance status of the selected component, optionallywithin the given number of future flights, as one of:

-   -   a positive category if an average of the weighted positive        scores of the models of the ensemble of related models is        greater than a threshold value and an average of the weighted        negative scores of the models of the ensemble of related models        is less than or equal to the threshold value;    -   a negative category if the average of the weighted positive        scores is less than or equal to the threshold value and the        average of the weighted negative scores is greater than the        threshold value; or    -   an unclassified category if both of the average of the weighted        positive scores and the average of the weighted negative scores        are greater than the threshold value or both are less than or        equal to the threshold value.

A1.1. The method of paragraph A1, wherein the average of the weightedpositive scores of the models of the ensemble of related models is anaverage of the weighted positive scores of a subset, optionally all, ofthe models of the ensemble of related models.

A1.2. The method of any of paragraphs A1-A1.1, wherein the average ofthe weighted negative scores of the models of the ensemble of relatedmodels is an average of the weighted negative scores of a subset of,optionally all, of the models of the ensemble of related models.

A2. The method of any of paragraphs A1-A1.2, wherein each model has aunique ROC among the ensemble of related models.

A3. The method of any of paragraphs A1-A2, further comprising selectingthe models of the ensemble of related models from a group of candidatemodels each having an ROC, wherein the models of the ensemble of relatedmodels are along a convex ROC hull of the group of candidate models.

A3.1. The method of paragraph A3, wherein the models of the ensemble ofrelated models are dispersed along the convex ROC hull of the group ofcandidate models.

A3.2. The method of any of paragraphs A3-A3.1, wherein the models of theensemble of related models are uniformly distributed along the convexROC hull of the group of candidate models.

A4. The method of any of paragraphs A1-A3.2, wherein at least one modelof the ensemble of related models is class biased.

A4.1. The method of paragraph A4, wherein the ensemble of related modelsincludes a model that has a positive class bias and a model that has anegative class bias.

A5. The method of any of paragraphs A1-A4.1, wherein the positive scoreand the negative score of at least one model, optionally of all models,relate respectively to a first category of a binary classification and asecond category of the binary classification.

A6. The method of any of paragraphs A1-A5, wherein at least one,optionally each, of the models of the ensemble of related models is aclassifier.

A7. The method of any of paragraphs A1-A6, wherein each model is theresult of guided machine learning.

A8. The method of any of paragraphs A1-A7, wherein at least one,optionally each, model includes, optionally is, at least one of a naiveBayes classifier, a support vector machine, a learned decision tree, anensemble of learned decision trees, or a neural network.

A9. The method of any of paragraphs A1-A8, wherein at least one,optionally each, model includes at least one of a statisticalcorrelation or a regression.

A10. The method of any of paragraphs A1-A9, wherein the positive scoreof each model indicates a likelihood of non-performance of the selectedcomponent and wherein the complementary negative score of each modelindicates a likelihood of continued performance of the selectedcomponent.

A11. The method of any of paragraphs A1-A10, wherein the weightedpositive score for each model is a product of the positive score of therespective model and an inverse exponential with an exponentproportional to the false positive rate of the respective model.

A12. The method of any of paragraphs A1-A11, wherein the weightednegative score for each model is a product of the negative score of therespective model and an inverse exponential with an exponentproportional to the false negative rate of the respective model.

A13. The method of any of paragraphs A1-A12, further comprisingdetermining a likelihood of non-performance of the selected component,optionally within the given number of future flights, based on theaverage of the weighted positive scores and/or the average of theweighted negative scores.

A14. The method of any of paragraphs A1-A13, further comprisingdetermining a confidence level in the performance status of the selectedcomponent based on the average of the weighted positive scores and/orthe average of the weighted negative scores.

A15. The method of any of paragraphs A1-A14, wherein the positive scorefor each model indicates an impending non-performance event of theselected component, optionally within the given number of futureflights.

A16. The method of any of paragraphs A1-A15, wherein the negative scorefor each model indicates no impending non-performance event of theselected component, optionally within the given number of futureflights.

A17. The method of any of paragraphs A1-A16, wherein the flight data wascollected during a series of flights.

A18. The method of any of paragraphs A1-A17, further comprisingcollecting the flight data during a flight or series of flights of theaircraft.

A18.1. The method of paragraph A18, wherein the collecting includescollecting the flight data with a sensor on board the aircraft.

A19. The method of any of paragraphs A1-A18.1, wherein the flight datawas collected with a sensor on board the aircraft.

A20. The method of any of paragraphs A1-A19, wherein the extractingincludes determining a statistic of flight data during a time window,and optionally wherein the flight data includes sensor values and/orcontrol input values.

A20.1. The method of paragraph A20, wherein the statistic includes,optionally is, at least one of a minimum, a maximum, an average, amoving average, a variance, a skewness, a kurtosis, a deviation, acumulative value, a rate of change, or an average rate of change.

A20.2. The method of any of paragraphs A20-A20.1, wherein the statisticincludes, optionally is, at least one of a total number of data points,a maximum number of sequential data points, a minimum number ofsequential data points, an average number of sequential data points, anaggregate time, a maximum time, a minimum time, and/or an average timethat the sensor values are above, below, or about equal to a thresholdvalue.

A20.3. The method of any of paragraphs A20-A20.2, wherein the timewindow includes, optionally is, a duration of the flight, a portion of aduration of the flight, and/or a period of time including one or moreflights of the aircraft, and optionally wherein the time window includesa duration of each of a/the series of flights.

A20.4. The method of any of paragraphs A20-A20.3, wherein the timewindow is at most 10 seconds or at most 1 second.

A20.5. The method of any of paragraphs A20-A20.4, wherein the sensorvalues include at least one of an airspeed, a strain, a temperature, avoltage, a current, an ambient temperature, an ambient pressure, anacceleration, a vertical acceleration, a velocity, a vertical velocity,a pitch rate, a roll rate, a yaw rate, an angle of attack, an attitude,an altitude, a heading, or a component position.

A20.6. The method of any of paragraphs A20-A20.5, wherein the sensorvalues include a first sensor value relating to the selected componentand a second sensor value relating to another component of the aircraft.

A21. The method of any of paragraphs A1-A20.6, wherein the extractingincludes determining a difference of sensor values and/or control inputvalues during a time window and optionally wherein the flight dataincludes the sensor values and/or the control input values.

A21.1. The method of paragraph A21, wherein the time window includes,optionally is, a duration of the flight, a portion of a duration of theflight, and/or a period of time including one or more flights of theaircraft, and optionally wherein the time window includes a duration ofeach of a/the series of flights.

A21.2. The method of any of paragraphs A21-A21.1, wherein the timewindow is at most 10 seconds or at most 1 second.

A21.3. The method of any of paragraphs A21-A21.2, wherein the sensorvalues include at least one of an airspeed, a strain, a temperature, avoltage, a current, an ambient temperature, an ambient pressure, anacceleration, a vertical acceleration, a velocity, a vertical velocity,a pitch rate, a roll rate, a yaw rate, an angle of attack, an attitude,an altitude, a heading, or a component position.

A21.4. The method of any of paragraphs A21-A21.3, wherein the sensorvalues include a first sensor value relating to the selected componentand a second sensor value relating to another component of the aircraft.

A21.5. The method of any of paragraphs A21-A21.4, wherein the extractingincludes determining a statistic of the difference of sensor valuesand/or control input values during the time window.

A22. The method of any of paragraphs A1-A21.5, wherein the extractingincludes determining a difference between a first sensor or controlinput value and a second sensor or control input value, and wherein theflight data includes the first sensor or control input value and thesecond sensor or control input value.

A22.1. The method of paragraph A22, wherein the first sensor or controlinput value is either a first sensor value or a first control inputvalue.

A22.2. The method of any of paragraphs A22-A22.1, wherein the secondsensor or control input value is either a second sensor value or asecond control input value.

A22.3. The method of any of paragraphs A22-A22.2, wherein the firstsensor or control input value and the second sensor or control inputvalue relate to a sensed parameter, and optionally wherein the sensedparameter is selected from the group of a rate, a velocity, anacceleration, a position, a pressure, a temperature, a speed, a strain,a voltage, and a current.

A22.4. The method of any of paragraphs A22-A22.3, wherein the firstsensor or control input value and the second sensor or control inputvalue are measured at different locations.

A22.5. The method of any of paragraphs A22-A22.4, wherein the firstsensor or control input value and the second sensor or control inputvalue are measured at different points in time.

A22.6. The method of any of paragraphs A22-A22.5, wherein the firstsensor or control input value relates to the selected component and thesecond sensor or control input value relates to another component of theaircraft.

A22.7. The method of any of paragraphs A22-A22.6, wherein the extractingincludes determining a statistic of the difference between the firstsensor or control input value and the second sensor or control inputvalue.

A23. The method of any of paragraphs A1-A22.7, wherein the feature dataincludes at least one derived signal selected from a minimum of a sensorvalue, a maximum of a sensor value, an average of a sensor value, amoving average of a sensor value, a variance of a sensor value, askewness of a sensor value, a kurtosis of a sensor value, a deviation ofa sensor value, a cumulative value of a sensor value, a difference ofsensor values, a minimum of a control input value, a maximum of acontrol input value, an average of a control input value, a movingaverage of a control input value, a variance of a control input value, askewness of a control input value, a kurtosis of a control input value,a deviation of a control input value, a cumulative value of a controlinput value, a difference of control input values, or a statistic of anyof the above derived signals.

A24. The method of any of paragraphs A1-A23, further comprisingdisplaying the performance status by visual, audio, and/or tactiledisplay.

A25. The method of any of paragraphs A1-A24, further comprisingsignaling by visual, audio, and/or tactile indicator that the selectedcomponent is likely to have a non-performance event.

A26. A computerized system comprising:

a computer-readable memory;

a processing unit operatively coupled to the computer-readable memory;and

a computer-readable storage media assemblage, wherein the storage mediaassemblage is operatively coupled to the computer-readable memory andincludes instructions, that when executed by the processing unit, causethe system to perform the method of any of paragraphs A1-A25.

A27. A method of preventive maintenance for an aircraft, the methodcomprising:

performing the method of any of paragraphs A1-A25, wherein determiningthe performance status includes determining the performance status is apositive category; and

repairing the selected component before the given number of futureflights.

B1. A system for determining a performance status of a selectedcomponent in an aircraft, the system comprising:

a feature extraction module configured to extract feature data fromflight data collected during a flight of the aircraft, wherein thefeature data relates to performance of one or more components of theaircraft;

a performance classification module configured to determine aperformance status of the selected component of the aircraft, optionallywithin a given number of future flights of the aircraft, based on thefeature data, wherein the performance classification module isprogrammed to:

-   -   apply an ensemble of related models to produce, for each model,        a positive score and a complementary negative score related to        performance of the selected component, optionally within the        given number of future flights, wherein each model has an ROC        (receiver operating characteristic) and is characterized by a        false positive rate and a false negative rate;    -   weight the positive score of each model to produce a weighted        positive score for each model based on the false positive rate        of that model such that the weighted positive score of each        model is anti-correlated with the false positive rate of that        model;    -   weight the negative score of each model to produce a weighted        negative score for each model based on the false negative rate        of that model such that the weighted negative score of each        model is anti-correlated with the false negative rate of that        model; and    -   determine the performance status of the selected component,        optionally within the given number of future flights, as one of:        -   a positive category if an average of the weighted positive            scores of all of the models of the ensemble of related            models is greater than a threshold value and an average of            the weighted negative scores of all of the models of the            ensemble of related models is less than or equal to the            threshold value;        -   a negative category if the average of the weighted positive            scores is less than or equal to the threshold value and the            average of the weighted negative scores is greater than the            threshold value; or        -   an unclassified category if both of the average of the            weighted positive scores and the average of the weighted            negative scores are greater than the threshold value or both            are less than or equal to the threshold value.

B1.1. The system of paragraph B1, wherein the system is a system fordetermining impending component non-performance in the aircraft.

B2. The system of any of paragraphs B1-B1.1, wherein each model has aunique ROC among the ensemble of related models.

B3. The system of any of paragraphs B1-B2, wherein the models of theensemble of related models are selected from a group of candidate modelseach having an ROC, wherein the models of the ensemble of related modelsare along a convex ROC hull of the group of candidate models.

B3.1. The system of paragraph B3, wherein the models of the ensemble ofrelated models are dispersed along the convex ROC hull of the group ofcandidate models.

B3.2. The system of any of paragraphs B3-B3.1, wherein the models of theensemble of related models are uniformly distributed along the convexROC hull of the group of candidate models.

B4. The system of any of paragraphs B1-B3.2, wherein at least one modelof the ensemble of related models is class biased.

B4.1. The system of paragraph B4, wherein the ensemble of related modelsincludes a model that has a positive class bias and a model that has anegative class bias.

B5. The system of any of paragraphs B1-B4.1, wherein the positive scoreand the negative score of at least one model, optionally of all models,relate respectively to a first category of a binary classification and asecond category of the binary classification.

B6. The system of any of paragraphs B1-B5, wherein at least one,optionally each, of the models of the ensemble of related models is aclassifier.

B7. The system of any of paragraphs B1-B6, wherein each model is theresult of guided machine learning.

B8. The system of any of paragraphs B1-B7, wherein at least one,optionally each, model includes, optionally is, at least one of a naiveBayes classifier, a support vector machine, a learned decision tree, anensemble of learned decision trees, or a neural network.

B9. The system of any of paragraphs B1-B8, wherein at least one,optionally each, model includes at least one of a statisticalcorrelation or a regression.

B10. The system of any of paragraphs B1-B9, wherein the positive scoreof each model indicates a likelihood of non-performance of the selectedcomponent and wherein the complementary negative score of each modelindicates a likelihood of continued performance of the selectedcomponent.

B11. The system of any of paragraphs B1-B10, wherein the weightedpositive score is a product of the positive score of the respectivemodel and an inverse exponential with an exponent proportional to thefalse positive rate of the respective model.

B12. The system of any of paragraphs B1-B11, wherein the weightednegative score is a product of the negative score of the respectivemodel and an inverse exponential with an exponent proportional to thefalse negative rate of the respective model.

B13. The system of any of paragraphs B1-B12, wherein the performanceclassification module is further programmed to determine a likelihood ofnon-performance of the selected component, optionally within the givennumber of future flights, based on the average of the weighted positivescores and/or the average of the weighted negative scores.

B14. The system of any of paragraphs B1-B13, wherein the performanceclassification module is further programmed to determine a confidencelevel in the performance status of the selected component based on theaverage of the weighted positive scores and/or the average of theweighted negative scores.

B15. The system of any of paragraphs B1-B14, wherein the positive scorefor each model indicates an impending non-performance event of theselected component, optionally within the given number of futureflights.

B16. The system of any of paragraphs B1-B15, wherein the negative scorefor each model indicates no impending non-performance event of theselected component, optionally within the given number of futureflights.

B17. The system of any of paragraphs B1-B16, further comprising a datalink configured to communicate with a flight data storage system, andoptionally wherein the flight data storage system is on board theaircraft.

B18. The system of any of paragraphs B1-B17, further comprising adisplay, wherein the display is configured to indicate the performancestatus with a visual, audio, and/or tactile display.

B19. The system of any of paragraphs B1-B18, further comprising a sensoron board the aircraft, wherein the sensor is configured to collectflight data during the flight of the aircraft.

B20. The system of any of paragraphs B1-B19, wherein the featureextraction module is configured to determine a statistic of flight dataduring a time window, and optionally wherein the flight data includessensor values and/or control input values.

B20.1. The system of paragraph B20, wherein the statistic includes,optionally is, at least one of a minimum, a maximum, an average, amoving average, a variance, a skewness, a kurtosis, a deviation, acumulative value, a rate of change, or an average rate of change.

B20.2. The system of any of paragraphs B20-B20.1, wherein the statisticincludes, optionally is, at least one of a total number of data points,a maximum number of sequential data points, a minimum number ofsequential data points, an average number of sequential data points, anaggregate time, a maximum time, a minimum time, or an average time thatthe sensor values are above, below, or about equal to a threshold value.

B20.3. The system of any of paragraphs B20-B20.2, wherein the timewindow includes, optionally is, a duration of the flight, a portion of aduration of the flight, and/or a period of time including one or moreflights of the aircraft.

B20.4. The system of any of paragraphs B20-B20.3, wherein the timewindow is at most 10 seconds or at most 1 second.

B20.5. The system of any of paragraphs B20-B20.4, wherein the sensorvalues include at least one of an airspeed, a strain, a temperature, avoltage, a current, an ambient temperature, an ambient pressure, anacceleration, a vertical acceleration, a velocity, a vertical velocity,a pitch rate, a roll rate, a yaw rate, an angle of attack, an attitude,an altitude, a heading, or a component position.

B20.6. The system of any of paragraphs B20-B20.5, wherein the sensorvalues include a first sensor value relating to the selected componentand a second sensor value relating to another component of the aircraft.

B20.7. The system of any of paragraphs B20-B20.6, wherein the controlinput values include a first control input value relating to theselected component and a second control input value relating to anothercomponent of the aircraft.

B21. The system of any of paragraphs B1-B20.7, wherein the featureextraction module is configured to determine a difference of sensorvalues and/or control input values during a time window and optionallywherein the flight data includes the sensor values and/or the controlinput values.

B21.1. The system of paragraph B21, wherein the time window includes,optionally is, a duration of the flight, a portion of a duration of theflight, and/or a period of time including one or more flights of theaircraft.

B21.2. The system of any of paragraphs B21-B21.1, wherein the timewindow is at most 10 seconds or at most 1 second.

B21.3. The system of any of paragraphs B21-B21.2, wherein the sensorvalues include at least one of an airspeed, a strain, a temperature, avoltage, a current, an ambient temperature, an ambient pressure, anacceleration, a vertical acceleration, a velocity, a vertical velocity,a pitch rate, a roll rate, a yaw rate, an angle of attack, an attitude,an altitude, a heading, or a component position.

B21.4. The system of any of paragraphs B21-B21.3, wherein the sensorvalues include a first sensor value relating to the selected componentand a second sensor value relating to another component of the aircraft.

B21.5. The system of any of paragraphs B21-B21.4, wherein the controlinput values include a first control input value relating to theselected component and a second control input value relating to anothercomponent of the aircraft.

B21.6. The system of any of paragraphs B21-B21.5, wherein the featureextraction module is configured to determine a statistic of thedifference of sensor values and/or control input values during the timewindow.

B22. The system of any of paragraphs B1-B21.6, wherein the featureextraction module is configured to determine a difference between afirst sensor or control input value and a second sensor or control inputvalue, and wherein the flight data includes the first sensor or controlinput value and the second sensor or control input value.

B22.1. The system of paragraph B22, wherein the first sensor or controlinput value is either a first sensor value or a first control inputvalue.

B22.2. The system of any of paragraphs B22-B22.1, wherein the secondsensor or control input value is either a second sensor value or asecond control input value.

B22.3. The system of any of paragraphs B22-B22.2, wherein the firstsensor or control input value and the second sensor or control inputvalue relate to a sensed parameter, and optionally wherein the sensedparameter is selected from the group of a rate, a velocity, anacceleration, a position, a pressure, a temperature, a speed, a strain,a voltage, and a current.

B22.4. The system of any of paragraphs B22-B22.3, wherein the firstsensor or control input value and the second sensor or control inputvalue are measured at different locations.

B22.5. The system of any of paragraphs B22-B22.4, wherein the firstsensor or control input value and the second sensor or control inputvalue are measured simultaneously or at different points in time.

B22.6. The system of any of paragraphs B22-B22.5, wherein the firstsensor or control input value relates to the selected component and thesecond sensor or control input value relates to another component of theaircraft.

B22.7. The system of any of paragraphs B22-B22.6, wherein the featureextraction module is configured to determine a statistic of thedifference between the first sensor value or control input value and thesecond sensor or control input value.

B23. The system of any of paragraphs B1-B22.7, further comprising:

a computer-readable memory;

a processing unit operatively coupled to the computer-readable memory;and

a computer-readable storage media assemblage, wherein the storage mediaassemblage is operatively coupled to the computer-readable memory andincludes the feature extraction module and the performanceclassification module.

As used herein, the terms “adapted” and “configured” mean that theelement, component, or other subject matter is designed and/or intendedto perform a given function. Thus, the use of the terms “adapted” and“configured” should not be construed to mean that a given element,component, or other subject matter is simply “capable of” performing agiven function but that the element, component, and/or other subjectmatter is specifically selected, created, implemented, utilized,programmed, and/or designed for the purpose of performing the function.It is also within the scope of the present disclosure that elements,components, and/or other recited subject matter that is recited as beingadapted to perform a particular function may additionally oralternatively be described as being configured to perform that function,and vice versa. Similarly, subject matter that is recited as beingconfigured to perform a particular function may additionally oralternatively be described as being operative to perform that function.

As used herein, the phrase, “for example,” the phrase, “as an example,”and/or simply the term “example,” when used with reference to one ormore components, features, details, structures, embodiments, and/ormethods according to the present disclosure, are intended to convey thatthe described component, feature, detail, structure, embodiment, and/ormethod is an illustrative, non-exclusive example of components,features, details, structures, embodiments, and/or methods according tothe present disclosure. Thus, the described component, feature, detail,structure, embodiment, and/or method is not intended to be limiting,required, or exclusive/exhaustive; and other components, features,details, structures, embodiments, and/or methods, including structurallyand/or functionally similar and/or equivalent components, features,details, structures, embodiments, and/or methods, are also within thescope of the present disclosure.

As used herein, the phrases “at least one of” and “one or more of,” inreference to a list of more than one entity, means any one or more ofthe entities in the list of entities, and is not limited to at least oneof each and every entity specifically listed within the list ofentities. For example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently, “at least one of A and/or B”)may refer to A alone, B alone, or the combination of A and B.

As used herein, the singular forms “a”, “an” and “the” may be intendedto include the plural forms as well, unless the context clearlyindicates otherwise.

The various disclosed elements of systems and apparatuses and steps ofmethods disclosed herein are not required of all systems, apparatuses,and methods according to the present disclosure, and the presentdisclosure includes all novel and non-obvious combinations andsubcombinations of the various elements and steps disclosed herein.Moreover, any of the various elements and steps, or any combination ofthe various elements and/or steps, disclosed herein may defineindependent inventive subject matter that is separate and apart from thewhole of a disclosed system, apparatus, or method. Accordingly, suchinventive subject matter is not required to be associated with thespecific systems, apparatuses, and methods that are expressly disclosedherein, and such inventive subject matter may find utility in systems,apparatuses, and/or methods that are not expressly disclosed herein.

1. A method of determining a performance status of a selected componentin an aircraft, the method comprising: extracting feature data fromflight data collected during a flight of the aircraft, wherein thefeature data relates to performance of one or more components of theaircraft; applying an ensemble of related models to produce, for eachmodel, a positive score and a complementary negative score related toperformance of the selected component, wherein each model ischaracterized by a false positive rate and a false negative rate;weighting the positive score of each model to produce a weightedpositive score for each model based on the false positive rate of thatmodel such that the weighted positive score of each model isanti-correlated with the false positive rate of that model; weightingthe negative score of each model to produce a weighted negative scorefor each model based on the false negative rate of that model such thatthe weighted negative score of each model is anti-correlated with thefalse negative rate of that model; and determining the performancestatus of the selected component as one of: a positive category if anaverage of the weighted positive scores of the models of the ensemble ofrelated models is greater than a threshold value and an average of theweighted negative scores of the models of the ensemble of related modelsis less than or equal to the threshold value; a negative category if theaverage of the weighted positive scores is less than or equal to thethreshold value and the average of the weighted negative scores isgreater than the threshold value; or an unclassified category if both ofthe average of the weighted positive scores and the average of theweighted negative scores are greater than the threshold value or bothare less than or equal to the threshold value.
 2. The method of claim 1,further comprising selecting the models of the ensemble of relatedmodels from a group of candidate models each having an ROC (receiveroperating characteristic), wherein the models of the ensemble of relatedmodels are along a convex ROC hull of the group of candidate models. 3.The method of claim 2, wherein the models of the ensemble of relatedmodels are dispersed along the convex ROC hull of the group of candidatemodels.
 4. The method of claim 1, wherein the ensemble of related modelsincludes a model that has a positive class bias and a model that has anegative class bias.
 5. The method of claim 1, wherein each modelincludes at least one of a naive Bayes classifier, a support vectormachine, a learned decision tree, an ensemble of learned decision trees,or a neural network.
 6. The method of claim 1, wherein the weightedpositive score for each model is a product of the positive score of therespective model and an inverse exponential with an exponentproportional to the false positive rate of the respective model.
 7. Themethod of claim 1, wherein the weighted negative score for each model isa product of the negative score of the respective model and an inverseexponential with an exponent proportional to the false negative rate ofthe respective model.
 8. The method of claim 1, further comprisingdetermining a likelihood of non-performance of the selected componentbased on the average of the weighted positive scores and the average ofthe weighted negative scores.
 9. The method of claim 1, furthercomprising collecting the flight data during a series of flights of theaircraft.
 10. The method of claim 1, further comprising signaling byvisual, audio, and/or tactile indicator that the selected component islikely to have a non-performance event.
 11. A computerized systemcomprising: a computer-readable memory; a processing unit operativelycoupled to the computer-readable memory; and a computer-readable storagemedia assemblage, wherein the storage media assemblage is operativelycoupled to the computer-readable memory and includes instructions, thatwhen executed by the processing unit, cause the system to: extractfeature data from flight data collected during a flight of the aircraft,wherein the feature data relates to performance of one or morecomponents of the aircraft; apply an ensemble of related models toproduce, for each model, a positive score and a complementary negativescore related to performance of the selected component, wherein eachmodel is characterized by a false positive rate and a false negativerate; weight the positive score of each model to produce a weightedpositive score for each model based on the false positive rate of thatmodel such that the weighted positive score of each model isanti-correlated with the false positive rate of that model; weight thenegative score of each model to produce a weighted negative score foreach model based on the false negative rate of that model such that theweighted negative score of each model is anti-correlated with the falsenegative rate of that model; and determine a performance status of theselected component as one of: a positive category if an average of theweighted positive scores of the models of the ensemble of related modelsis greater than a threshold value and an average of the weightednegative scores of the models of the ensemble of related models is lessthan or equal to the threshold value; a negative category if the averageof the weighted positive scores is less than or equal to the thresholdvalue and the average of the weighted negative scores is greater thanthe threshold value; or an unclassified category if both of the averageof the weighted positive scores and the average of the weighted negativescores are greater than the threshold value or both are less than orequal to the threshold value.
 12. A system for determining a performancestatus of a selected component in an aircraft, the system comprising: afeature extraction module configured to extract feature data from flightdata collected during a flight of the aircraft, wherein the feature datarelates to performance of one or more components of the aircraft; aperformance classification module configured to determine a performancestatus of the selected component of the aircraft based on the featuredata, wherein the performance classification module is programmed to:apply an ensemble of related models to produce, for each model, apositive score and a complementary negative score related to performanceof the selected component, wherein each model has an ROC (receiveroperating characteristic) characterized by a false positive rate and afalse negative rate; weight the positive score of each model to producea weighted positive score for each model based on the false positiverate of that model such that the weighted positive score of each modelis anti-correlated with the false positive rate of that model; weightthe negative score of each model to produce a weighted negative scorefor each model based on the false negative rate of that model such thatthe weighted negative score of each model is anti-correlated with thefalse negative rate of that model; and determine the performance statusof the selected component as one of: a positive category if an averageof the weighted positive scores of the models of the ensemble of relatedmodels is greater than a threshold value and an average of the weightednegative scores of the models of the ensemble of related models is lessthan or equal to the threshold value; a negative category if the averageof the weighted positive scores is less than or equal to the thresholdvalue and the average of the weighted negative scores is greater thanthe threshold value; or an unclassified category if both of the averageof the weighted positive scores and the average of the weighted negativescores are greater than the threshold value or both are less than orequal to the threshold value.
 13. The system of claim 12, wherein themodels of the ensemble of related models are selected from a group ofcandidate models each having an ROC, wherein the models of the ensembleof related models are along a convex ROC hull of the group of candidatemodels.
 14. The system of claim 12, wherein at least one model of theensemble of related models is class biased.
 15. The system of claim 12,wherein the positive score of each model indicates a likelihood ofnon-performance of the selected component and wherein the complementarynegative score of each model indicates a likelihood of continuedperformance of the selected component.
 16. The system of claim 12,wherein the weighted positive score is a product of the positive scoreof the respective model and an inverse exponential with an exponentproportional to the false positive rate of the respective model.
 17. Thesystem of claim 12, wherein the weighted negative score is a product ofthe negative score of the respective model and an inverse exponentialwith an exponent proportional to the false negative rate of therespective model.
 18. The system of claim 12, further comprising a datalink configured to communicate with a flight data storage system. 19.The system of claim 12, further comprising a display, wherein thedisplay is configured to indicate the performance status with at leastone of a visual, audio, or tactile display.
 20. The system of claim 12,further comprising a sensor on board the aircraft, wherein the sensor isconfigured to collect flight data during the flight of the aircraft.